{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright 2020 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/GoogleCloudPlatform/keras-idiomatic-programmer/blob/master/books/deep-learning-design-patterns/Workshops/Novice/Deep%20Learning%20Design%20Patterns%20-%20Workshop%20-%20Chapter%20I.ipynb\">\n",
    "<img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Learning Design Patterns - Code Labs \n",
    "\n",
    "## Lab Exercise #1 - Get Started with a Deep Neural Network (DNN)\n",
    "\n",
    "## Prerequistes:\n",
    "\n",
    "    1. Familiar with Python\n",
    "    2. Completed Chapter I: Neural Networks\n",
    "    \n",
    "## Objectives:\n",
    "\n",
    "    1. Install TF.Keras related packages\n",
    "    2. Create a basic DNN (input layer, hidden layer, output layer)\n",
    "        A. Correctly set the input shape.\n",
    "        B. Correctly set the number of nodes.\n",
    "        C. Correctly set activation function.\n",
    "        D. Practive using various Sequential API styles and the Functional API.\n",
    "    3. Review a model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup:\n",
    "\n",
    "Install relevant packages to get started with Keras, and then import them.\n",
    "\n",
    "For Tensorflow 2.0+ install, you may get incompatibility errors with Tensorboard and TFX. These can be ignored for the purpose of this lab exercise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install numpy math library\n",
    "!pip install -U numpy\n",
    "# Install Tensorflow+Keras\n",
    "!pip install -U tensorflow\n",
    "\n",
    "# Import keras and TF\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "\n",
    "print(np.__version__)              # should be 1.17+\n",
    "print(tf.__version__)              # should be 2.0+"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic DNN as Sequential API (long form)\n",
    "\n",
    "Let's use the long form of a Sequential method. That means that we will use the add() method for each step.\n",
    "\n",
    "Below is a nearly complete DNN for a multi-class classifier. You fill in the blanks (replace the ??), make sure it passes the Python interpreter, and then verify it's correctness with the summary output.\n",
    "\n",
    "You will need to:\n",
    "\n",
    "    1. Fill in the input shape for the 28x28 image input.\n",
    "    2. Specify number of nodes for the dense layers.\n",
    "    3. Set the activation function for the hidden dense layers and ouput dense layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import Sequential\n",
    "from tensorflow.keras.layers import Flatten, Dense, Activation\n",
    "\n",
    "# define the input shape for a 28x28 grayscale image (like MNIST)\n",
    "# HINT: should be a tuple of height and width\n",
    "input_shape=( ?? )\n",
    "\n",
    "# Let's start a sequential model\n",
    "model = Sequential()\n",
    "\n",
    "# Let's flatten the 28x28 image to a 784 1D vector\n",
    "model.add(Flatten(input_shape=input_shape))\n",
    "\n",
    "# Let's add the input layer as a dense layer of 128 nodes\n",
    "# HINT: the parameter is the number of nodes\n",
    "model.add(Dense(??))\n",
    "\n",
    "# Add the activation function\n",
    "# HINT: use the best practice convention for a non-output Dense layer\n",
    "model.add(Activation(??))\n",
    "\n",
    "# Add the hidden layer with 512 nodes\n",
    "model.add(Dense(??))\n",
    "model.add(Activation(??))\n",
    "\n",
    "# Add the output layer with 10 nodes\n",
    "model.add(Dense(??))\n",
    "\n",
    "# Add the activation function\n",
    "# HINT: use the best practice for a multi-class classifier\n",
    "model.add(Activation(??))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Verify the model architecture using summary method\n",
    "\n",
    "It should look like below:\n",
    "\n",
    "```\n",
    "Layer (type)                 Output Shape              Param #   \n",
    "=================================================================\n",
    "flatten_1 (Flatten)          (None, 784)               0         \n",
    "_________________________________________________________________\n",
    "dense_4 (Dense)              (None, 128)               100480    \n",
    "_________________________________________________________________\n",
    "activation_4 (Activation)    (None, 128)               0         \n",
    "_________________________________________________________________\n",
    "dense_5 (Dense)              (None, 512)               66048     \n",
    "_________________________________________________________________\n",
    "activation_5 (Activation)    (None, 512)               0         \n",
    "_________________________________________________________________\n",
    "dense_6 (Dense)              (None, 10)                5130      \n",
    "_________________________________________________________________\n",
    "activation_6 (Activation)    (None, 10)                0         \n",
    "=================================================================\n",
    "Total params: 171,658\n",
    "Trainable params: 171,658\n",
    "Non-trainable params: 0\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Basic DNN as Sequential API (short form)\n",
    "\n",
    "Let's repeat the above, but use the short form.\n",
    "\n",
    "You will need to:\n",
    "\n",
    "    1. Specify the activation functions as a parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import Sequential\n",
    "from tensorflow.keras.layers import Dense\n",
    "\n",
    "# Let's start a sequential model\n",
    "model = Sequential()\n",
    "\n",
    "# Let's add a first flattening layer to flatten the 28x28 image into 1D vector\n",
    "model.add(Flatten(input_shape=(28, 28, 1)))\n",
    "\n",
    "# Let's add the input layer as a dense layer of 128 nodes\n",
    "# HINT: use best practices for activation functions\n",
    "model.add(Dense(128, activation=??))\n",
    "\n",
    "# Add the hidden layer with 512 nodes\n",
    "model.add(Dense(512, activation=??))\n",
    "\n",
    "# Add the output layer with 10 nodes\n",
    "model.add(Dense(10, activation=??))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Verify the model architecture using summary method\n",
    "\n",
    "It should be identical to the previous. But note in this case, summary() does not show the activation functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic DNN as Sequential API (list form)\n",
    "\n",
    "Let's repeat the above, but instead specify the layers as a parameter to the Sequential() object, we will specify them using a list.\n",
    "\n",
    "You will need to:\n",
    "\n",
    "    1. Define the hidden layer.\n",
    "    2. Define the output (classifier) layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import Sequential\n",
    "from tensorflow.keras.layers import Dense\n",
    "\n",
    "# Let's start a sequential model\n",
    "model = Sequential([ # input layer\n",
    "                     Flatten(input_shape=(28, 28, 1)),\n",
    "                     Dense(128, activation='relu'),\n",
    "                     # hidden layer\n",
    "                     ??,\n",
    "                     # output layer\n",
    "                     ??,\n",
    "                    ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Verify the model architecture using summary method\n",
    "\n",
    "It should be identical to the previous."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic DNN as Functional API \n",
    "\n",
    "Let's repeat the above, but instead specify the layers using the functional API.\n",
    "\n",
    "This will need to:\n",
    "\n",
    "    1. Specify the input shape for the Input object\n",
    "    2. Connect the Input object to the Flatten layer.\n",
    "    3. Connect the Flatten output to the input Dense layer.\n",
    "    4. Specify the hidden and output layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import Model, Input\n",
    "from tensorflow.keras.layers import Dense, Flatten\n",
    "\n",
    "# Define the input vector for the 28x28 images\n",
    "# HINT: Should be a tuple of height and width\n",
    "inputs = Input(??)\n",
    "\n",
    "# Let's flatten the image into a 1D vector\n",
    "# HINT: the input vector is where you defined the input shape\n",
    "x = Flatten()(??)\n",
    "\n",
    "# Define the input layer and connect the flattened input vector\n",
    "# HINT: the output of Flatten() is the flattened input vector\n",
    "x = Dense(128, activation='relu')(??)\n",
    "\n",
    "# Define the hidden layer and connect the hidden layer to it.\n",
    "x = ??(512, activation='relu')(??)\n",
    "\n",
    "# Define the output layer and connect the hidden layer to it.\n",
    "outputs = ??(10, activation='softmax')(??)\n",
    "\n",
    "# Let's put it together: \n",
    "# HINT: inputs to outputs\n",
    "model = Model(inputs, outputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Verify the model architecture using summary method\n",
    "\n",
    "It should be identical to the previous."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training\n",
    "\n",
    "Finally, you will train the model --eventhough we haven't covered training yet. Just follow the steps and later we will revisit and go into details.\n",
    "\n",
    "We will make two copies of the model. One we will train without preprocessing (not covered yet) the MNIST dataset, and the other with preprocessing and compare the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def makeModel():\n",
    "    # Make a DNN model\n",
    "    inputs = Input((28, 28))\n",
    "    x = Flatten()(inputs)\n",
    "    x = Dense(128, activation='relu')(x)\n",
    "    x = Dense(512, activation='relu')(x)\n",
    "    outputs = Dense(10, activation='softmax')(x)\n",
    "    model = Model(inputs, outputs)\n",
    "    # Compile the model\n",
    "    model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['acc'])\n",
    "    return model\n",
    "\n",
    "# We make two copies of the model\n",
    "model_a = makeModel()\n",
    "model_b = makeModel()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset\n",
    "\n",
    "tf.Keras has a variety of builtin datasets, including the MNIST dataset. We will start by getting the dataset as non-processed data. This will include:\n",
    "\n",
    "    training data : *_train\n",
    "    test data .   : *_test\n",
    "    \n",
    "Both the training and test data consists of the training images (x_\\*) and corresponding labels (y_\\*)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.datasets import mnist\n",
    "\n",
    "# Load the dataset\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train w/o preprocessing the data\n",
    "\n",
    "Let's train the first model without preprocessing the data for 10 epochs (not covered yet). Pay attention to the end of the final (epoch 10) training accuracy (acc:) and validation accuracy (val_acc:)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_a.fit(x_train, y_train, epochs=10, batch_size=32, validation_split=0.1, verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training with preprocessed data.\n",
    "\n",
    "Let's preprocess the image data by normalizing the pixel values --not covered yet, and then train the second model. Pay attention to the final training and validation accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# Preprocess the image data\n",
    "x_train = (x_train / 255.0).astype(np.float32)\n",
    "\n",
    "# Train the model\n",
    "model_b.fit(x_train, y_train, epochs=10, batch_size=32, validation_split=0.1, verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Observation\n",
    "\n",
    "Note how by preprocessing the data, we get a higher accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## End of Lab"
   ]
  }
 ],
 "metadata": {
  "environment": {
   "name": "tf2-2-2-gpu.2-2.m50",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-2-2-gpu.2-2:m50"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
